{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recognize and Count Objects in a Video\n",
    "\n",
    "An object detection classifier can be used to identify and locate objects in a static image. When using video, you can use the same approach to static detection individual frames.  In this Jupyter Notebook, we'll use the [IBM PowerAI Vision](https://www.ibm.com/us-en/marketplace/ibm-powerai-vision) for object detection and [OpenCV Python API](https://opencv.org/) to process the video.\n",
    "\n",
    "Before running this notebook, you will need to train and deploy an object detection model. PowerAI Vision has auto-labeling to enhance your dataset for accuracy when using video input. After you train and deploy your model, set the `POWER_AI_VISION_API_URL` constant below to use your model for inference.\n",
    "\n",
    "Extracting frames and locating objects is easy with OpenCV and PowerAI Vision. The challenge is how to keep track of objects if you want to count them. As an object moves, you will need to be able to determine whether or not you have already counted the object. In this notebook, we'll use the OpenCV Tracking API to follow cars down the road while we run PowerAI Vision object detection on a sample of the frames. With tracking, we'll be able to avoid double counting without requiring a lot of code.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| First Detected... | Followed Down the Road |\n",
    "| :---: | :---: |\n",
    "| ![detected](https://raw.githubusercontent.com/IBM/powerai-counting-cars/master/doc/source/images/output-frame_00011.jpg) | ![tracked](https://raw.githubusercontent.com/IBM/powerai-counting-cars/master/doc/source/images/output-frame_00128.jpg) |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First setup some parameters\n",
    "\n",
    "### Required setup!\n",
    "\n",
    "Your PowerAI Vision API endpoint for the model that you trained and deployed will need to be set here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this URL to the API endpoint of your deployed model.\n",
    "POWER_AI_VISION_API_URL = \"https://ny1.ptopenlab.com/powerai-vision-ny/api/dlapis/your-guid-here\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional configuration\n",
    "\n",
    "Here you can customize some settings to tune your results.\n",
    "\n",
    "> NOTE: The notebook uses sampling and cached results to speed things up for iterative development. If you change the video, you will need to run with `CLEAN = True` to delete and regenerate your cached frames and inference results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLEAN = False  # USE WITH CARE! Wipe out saved files when this is true (else reuse for speed)\n",
    "input_video_url = \"https://raw.githubusercontent.com/IBM/powerai-counting-cars/master/data/test_video.mp4\"  # The input video\n",
    "START_LINE = 0  # If start line is > 0, cars won't be added until below the line (try 200)\n",
    "FRAMES_DIR = \"frames\"  # Output dir to hold/cache the original frames\n",
    "OUTPUT_DIR = \"output\"  # Output dir to hold the annotated frames\n",
    "SAMPLING = 10  # Classify every n frames (use tracking in between)\n",
    "CONFIDENCE = 0.80  # Confidence threshold to filter iffy objects\n",
    "\n",
    "# OpenCV colors are (B, G, R) tuples -- RGB in reverse\n",
    "WHITE = (255, 255, 255)\n",
    "YELLOW = (66, 244, 238)\n",
    "GREEN = (80, 220, 60)\n",
    "LIGHT_CYAN = (255, 255, 224)\n",
    "DARK_BLUE = (139, 0, 0)\n",
    "GRAY = (128, 128, 128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Python Requirements\n",
    "\n",
    "Install Python packages from pypi.org.\n",
    "We're pinning versions here to what was last tested.\n",
    "You might want to comment these out after you have the packages, or update the versions if you\n",
    "want to try the latest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install opencv-python==3.4.4.19\n",
    "!pip install opencv-contrib-python==3.4.4.19\n",
    "!pip install requests==2.21.0\n",
    "!pip install pandas==0.23.4\n",
    "!pip install urllib3==1.24.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import glob\n",
    "import math\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "import cv2\n",
    "from IPython.display import clear_output, Image, display\n",
    "import requests\n",
    "from requests.packages.urllib3.exceptions import InsecureRequestWarning\n",
    "requests.packages.urllib3.disable_warnings(InsecureRequestWarning)\n",
    "print(\"Warning: Certificates not verified!\")\n",
    "\n",
    "%matplotlib notebook\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the video\n",
    "This will download a small example video.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget {input_video_url}\n",
    "input_video = input_video_url.split('/')[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create or clean the directories\n",
    "Caching the frames and output directories allows the processing to continue where it left off. This is particularly useful when using a shared system with deployment time limits. This also allows you to quickly `Run all` when tweaking Python code that does not affect the inference.\n",
    "\n",
    "If you change the input video or just want a fresh start, you should `CLEAN` or change the directory names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if CLEAN:\n",
    "    if os.path.isdir(FRAMES_DIR):\n",
    "        shutil.rmtree(FRAMES_DIR)\n",
    "    if os.path.isdir(OUTPUT_DIR):\n",
    "        shutil.rmtree(OUTPUT_DIR)\n",
    "\n",
    "if not os.path.isdir(FRAMES_DIR):\n",
    "    os.mkdir(FRAMES_DIR)\n",
    "if not os.path.isdir(OUTPUT_DIR):\n",
    "    os.mkdir(OUTPUT_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse and explode the video file into JPEGs\n",
    "Each frame is saved as an individual JPEG file for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.isfile(input_video):\n",
    "    video_capture = cv2.VideoCapture(input_video)\n",
    "else:\n",
    "    raise Exception(\"File %s doesn't exist!\" % input_video)\n",
    "\n",
    "total_frames = int(video_capture.get(cv2.CAP_PROP_FRAME_COUNT))\n",
    "print(\"Frame count estimate is %d\" % total_frames)\n",
    "\n",
    "num = 0\n",
    "while video_capture.get(cv2.CAP_PROP_POS_FRAMES) < video_capture.get(cv2.CAP_PROP_FRAME_COUNT):\n",
    "    success, image = video_capture.read()\n",
    "    if success:\n",
    "        num = int(video_capture.get(cv2.CAP_PROP_POS_FRAMES))\n",
    "        print(\"Writing frame {num} of {total_frames}\".format(\n",
    "            num=num, total_frames=total_frames), end=\"\\r\")\n",
    "        cv2.imwrite('{frames_dir}/frame_{num:05d}.jpg'.format(\n",
    "            frames_dir=FRAMES_DIR, num=num), image)\n",
    "    else:\n",
    "        # TODO: If this happens, we need to add retry code\n",
    "        raise Exception('Error writing frame_{num:05d}.jpg'.format(\n",
    "            num=int(video_capture.get(cv2.CAP_PROP_POS_FRAMES))))\n",
    "\n",
    "print(\"\\nWrote {num} frames\".format(num=num))\n",
    "\n",
    "FRAME_FPS = int(video_capture.get(cv2.CAP_PROP_FPS))\n",
    "FRAME_WIDTH = int(video_capture.get(cv2.CAP_PROP_FRAME_WIDTH))\n",
    "FRAME_HEIGHT = int(video_capture.get(cv2.CAP_PROP_FRAME_HEIGHT))\n",
    "ROI_YMAX = int(round(FRAME_HEIGHT * 0.75))  # Bottom quarter = finish line\n",
    "\n",
    "print(\"Frame Dimensions: %sx%s\" % (FRAME_WIDTH, FRAME_HEIGHT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PowerAI Vision inference wrapper\n",
    "Define a helper/wrapper to call PowerAI Vision and return the inference result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "s = requests.Session()\n",
    "\n",
    "\n",
    "def detect_objects(filename):\n",
    "\n",
    "    with open(filename, 'rb') as f:\n",
    "        # WARNING! verify=False is here to allow an untrusted cert!\n",
    "        r = s.post(POWER_AI_VISION_API_URL,\n",
    "                   files={'files': (filename, f)},\n",
    "                   verify=False)\n",
    "\n",
    "    return r.status_code, json.loads(r.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the API on a single frame\n",
    "Let's look at the result of a single inference operation from the PowerAI Vision Object Detection API. We see a standard HTTP return code, and a JSON response which includes the image URL, and tuples that indicate the confidence and bounding-box coordinates of the objects that we classified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rc, jsonresp = detect_objects('frames/frame_00100.jpg')\n",
    "\n",
    "print(\"rc = %d\" % rc)\n",
    "print(\"jsonresp: %s\" % jsonresp)\n",
    "if 'classified' in jsonresp:\n",
    "    print(\"Got back %d objects\" % len(jsonresp['classified']))\n",
    "print(json.dumps(jsonresp, indent=2))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get object detection results for sampled frames\n",
    "Since we've stored all video frames on disk (for easy reference), we can iterate over those files\n",
    "and make queries as appropriate to PowerAI Vision's API. We'll store the results in a\n",
    "`tracking_results` dictionary, organized by file name. Since we are tracking objects from frame\n",
    "to frame, we can use sampling to decide how often to check for new objects.\n",
    "\n",
    "We're also caching the results so that you can change later code and run the notebook over\n",
    "without running the same inference over again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Serialize requests, storing them in a \"tracking_results\" dict\n",
    "\n",
    "try:\n",
    "    with open('frames/frame-data-newmodel.json') as existing_results:\n",
    "        tracking_results = json.load(existing_results)\n",
    "except Exception:\n",
    "    # Any fail to read existing results means we start over\n",
    "    tracking_results = {}\n",
    "\n",
    "print(\"Sampling every %sth frame\" % SAMPLING)\n",
    "i = 0\n",
    "cache_used = 0\n",
    "sampled = 0\n",
    "for filename in sorted(glob.glob('frames/frame_*.jpg')):\n",
    "    i += 1\n",
    "\n",
    "    if not i % SAMPLING == 0:  # Sample every Nth\n",
    "        continue\n",
    "\n",
    "    existing_result = tracking_results.get(filename)\n",
    "    if existing_result and existing_result['result'] == 'success':\n",
    "        cache_used += 1\n",
    "    else:\n",
    "        rc, results = detect_objects(filename)\n",
    "        if rc != 200 or results['result'] != 'success':\n",
    "            print(\"ERROR rc=%d for %s\" % (rc, filename))\n",
    "            print(\"ERROR result=%s\" % results)\n",
    "        else:\n",
    "            sampled += 1\n",
    "            # Save frequently to cache partial results\n",
    "            tracking_results[filename] = results\n",
    "            with open('frames/frame-data-newmodel.json', 'w') as fp:\n",
    "                json.dump(tracking_results, fp)\n",
    "\n",
    "    print(\"Processed file {num} of {total_frames} (used cache {cache_used} times)\".format(\n",
    "        num=i, total_frames=total_frames, cache_used=cache_used), end=\"\\r\")\n",
    "\n",
    "# Finally, write all our results\n",
    "with open('frames/frame-data-newmodel.json', 'w') as fp:\n",
    "    json.dump(tracking_results, fp)\n",
    "\n",
    "print(\"\\nDone\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define helper functions for tracking and drawing labels\n",
    "Refer to the [OpenCV docs.](https://docs.opencv.org/3.4.1/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def label_object(color, textcolor, fontface, image, car, textsize, thickness, xmax, xmid, xmin, ymax, ymid, ymin):\n",
    "    cv2.rectangle(image, (xmin, ymin), (xmax, ymax), color, thickness)\n",
    "    pos = (xmid - textsize[0]//2, ymid + textsize[1]//2)\n",
    "    cv2.putText(image, car, pos, fontface, 1, textcolor, thickness, cv2.LINE_AA)\n",
    "\n",
    "\n",
    "def update_trackers(image, counters):\n",
    "    left_lane = counters['left_lane']\n",
    "    right_lane = counters['right_lane']\n",
    "    boxes = []\n",
    "    color = (80, 220, 60)\n",
    "    fontface = cv2.FONT_HERSHEY_SIMPLEX\n",
    "    fontscale = 1\n",
    "    thickness = 1\n",
    "\n",
    "    for n, pair in enumerate(trackers):\n",
    "        tracker, car = pair\n",
    "        textsize, _baseline = cv2.getTextSize(\n",
    "            car, fontface, fontscale, thickness)\n",
    "        success, bbox = tracker.update(image)\n",
    "\n",
    "        if not success:\n",
    "            counters['lost_trackers'] += 1\n",
    "            del trackers[n]\n",
    "            continue\n",
    "\n",
    "        boxes.append(bbox)  # Return updated box list\n",
    "\n",
    "        xmin = int(bbox[0])\n",
    "        ymin = int(bbox[1])\n",
    "        xmax = int(bbox[0] + bbox[2])\n",
    "        ymax = int(bbox[1] + bbox[3])\n",
    "        xmid = int(round((xmin+xmax)/2))\n",
    "        ymid = int(round((ymin+ymax)/2))\n",
    "\n",
    "        if ymid >= ROI_YMAX:\n",
    "            label_object(WHITE, WHITE, fontface, image, car, textsize, 1, xmax, xmid, xmin, ymax, ymid, ymin)\n",
    "            # Count left-lane, right-lane as cars ymid crosses finish line\n",
    "            if xmid < 630:\n",
    "                left_lane += 1\n",
    "            else:\n",
    "                right_lane += 1\n",
    "            # Stop tracking cars when they hit finish line\n",
    "            del trackers[n]\n",
    "        else:\n",
    "            # Rectangle and number on the cars we are tracking\n",
    "            label_object(color, YELLOW, fontface, image, car, textsize, 4, xmax, xmid, xmin, ymax, ymid, ymin)\n",
    "\n",
    "    # Add finish line overlay/line\n",
    "    overlay = image.copy()\n",
    "\n",
    "    # Shade region of interest (ROI). We're really just using the top line.\n",
    "    cv2.rectangle(overlay,\n",
    "                  (0, ROI_YMAX),\n",
    "                  (FRAME_WIDTH, FRAME_HEIGHT), DARK_BLUE, cv2.FILLED)\n",
    "    cv2.addWeighted(overlay, 0.6, image, 0.4, 0, image)\n",
    "\n",
    "    # Draw start line, if > 0\n",
    "    if START_LINE > 0:\n",
    "        cv2.line(image, (0, START_LINE), (FRAME_WIDTH, START_LINE), GRAY, 4, cv2.LINE_AA)\n",
    "    # Draw finish line with lane hash marks\n",
    "    cv2.line(image, (0, ROI_YMAX), (FRAME_WIDTH, ROI_YMAX), LIGHT_CYAN, 4, cv2.LINE_AA)\n",
    "    cv2.line(image, (350, ROI_YMAX - 20), (350, ROI_YMAX + 20), LIGHT_CYAN, 4, cv2.LINE_AA)\n",
    "    cv2.line(image, (630, ROI_YMAX - 20), (630, ROI_YMAX + 20), LIGHT_CYAN, 4, cv2.LINE_AA)\n",
    "    cv2.line(image, (950, ROI_YMAX - 20), (950, ROI_YMAX + 20), LIGHT_CYAN, 4, cv2.LINE_AA)\n",
    "\n",
    "    # Add lane counter\n",
    "    cv2.putText(image, \"Lane counter:\", (30, ROI_YMAX + 80), fontface, 1.5, LIGHT_CYAN, 4, cv2.LINE_AA)\n",
    "    cv2.putText(image, str(left_lane), (480, ROI_YMAX + 80), fontface, 1.5, LIGHT_CYAN, 4, cv2.LINE_AA)\n",
    "    cv2.putText(image, str(right_lane), (800, ROI_YMAX + 80), fontface, 1.5, LIGHT_CYAN, 4, cv2.LINE_AA)\n",
    "    seconds = counters['frames'] / FRAME_FPS\n",
    "    cv2.putText(image, \"Cars/second:\", (35, ROI_YMAX + 110), fontface, 0.5, LIGHT_CYAN, 1, cv2.LINE_AA)\n",
    "    cv2.putText(image, '{0:.2f}'.format(left_lane / seconds), (480, ROI_YMAX + 110), fontface, 0.5, LIGHT_CYAN, 1, cv2.LINE_AA)\n",
    "    cv2.putText(image, '{0:.2f}'.format(right_lane / seconds), (800, ROI_YMAX + 110), fontface, 0.5, LIGHT_CYAN, 1, cv2.LINE_AA)\n",
    "\n",
    "    counters['left_lane'] = left_lane\n",
    "    counters['right_lane'] = right_lane\n",
    "    return boxes, counters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def not_tracked(objects, boxes):\n",
    "    if not objects:\n",
    "        return []  # No new classified objects to search for\n",
    "    if not boxes:\n",
    "        return objects  # No existing boxes, return all objects\n",
    "\n",
    "    new_objects = []\n",
    "    for obj in objects:\n",
    "        ymin = obj.get(\"ymin\", \"\")\n",
    "        ymax = obj.get(\"ymax\", \"\")\n",
    "        ymid = int(round((ymin+ymax)/2))\n",
    "        xmin = obj.get(\"xmin\", \"\")\n",
    "        xmax = obj.get(\"xmax\", \"\")\n",
    "        xmid = int(round((xmin+xmax)/2))\n",
    "        box_range = ((xmax - xmin) + (ymax - ymin)) / 2\n",
    "        for bbox in boxes:\n",
    "            bxmin = int(bbox[0])\n",
    "            bymin = int(bbox[1])\n",
    "            bxmax = int(bbox[0] + bbox[2])\n",
    "            bymax = int(bbox[1] + bbox[3])\n",
    "            bxmid = int((bxmin + bxmax) / 2)\n",
    "            bymid = int((bymin + bymax) / 2)\n",
    "            if math.sqrt((xmid - bxmid)**2 + (ymid - bymid)**2) < box_range:\n",
    "                # found existing, so break (do not add to new_objects)\n",
    "                break\n",
    "        else:\n",
    "            new_objects.append(obj)\n",
    "\n",
    "    return new_objects\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def in_range(obj):\n",
    "    ymin = obj['ymin']\n",
    "    ymax = obj['ymax']\n",
    "    if ymin < START_LINE or ymax > ROI_YMAX:\n",
    "        # Don't add new trackers before start or after finish.\n",
    "        # Start line can help avoid overlaps and tracker loss.\n",
    "        # Finish line protection avoids counting the car twice.\n",
    "        return False\n",
    "    return True\n",
    "    \n",
    "def add_new_object(obj, image, cars):\n",
    "    car = str(cars)\n",
    "    xmin = obj['xmin']\n",
    "    xmax = obj['xmax']\n",
    "    ymin = obj['ymin']\n",
    "    ymax = obj['ymax']\n",
    "    xmid = int(round((xmin+xmax)/2))\n",
    "    ymid = int(round((ymin+ymax)/2))\n",
    "    fontface = cv2.FONT_HERSHEY_SIMPLEX\n",
    "    fontscale = 1\n",
    "    thickness = 1\n",
    "    textsize, _baseline = cv2.getTextSize(\n",
    "        car, fontface, fontscale, thickness)\n",
    "\n",
    "    # init tracker\n",
    "    tracker = cv2.TrackerKCF_create()  # Note: Try comparing KCF with MIL\n",
    "    success = tracker.init(image, (xmin, ymin, xmax-xmin, ymax-ymin))\n",
    "    if success:\n",
    "        trackers.append((tracker, car))\n",
    "\n",
    "    label_object(GREEN, YELLOW, fontface, image, car, textsize, 4, xmax, xmid, xmin, ymax, ymid, ymin)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference, tracking, and annotation\n",
    "Loop through the saved frames and:\n",
    "1. Update the trackers to follow already detected objects from frame to frame.\n",
    "1. Look for new objects if we ran inference on this frame.\n",
    "    * Check for overlap with tracked objects.\n",
    "    * If no overlap, assign a sequence number and start tracking.\n",
    "1. Write an annotated image with tracked objects highlighted and numbered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cars = 0\n",
    "trackers = []\n",
    "counters = {\n",
    "    'left_lane':  0,\n",
    "    'right_lane':  0,\n",
    "    'lost_trackers': 0,\n",
    "    'frames': 0,\n",
    "}\n",
    "\n",
    "with open('frames/frame-data-newmodel.json') as existing_results:\n",
    "    tracking_results = json.load(existing_results)\n",
    "\n",
    "for filename in sorted(glob.glob('frames/frame_*.jpg')):\n",
    "    counters['frames'] += 1\n",
    "    img = cv2.imread(filename)\n",
    "    boxes, counters = update_trackers(img, counters)\n",
    "\n",
    "    if filename in tracking_results and 'classified' in tracking_results[filename]:\n",
    "        jsonresp = tracking_results[filename]\n",
    "        for obj in not_tracked(jsonresp['classified'], boxes):\n",
    "            if in_range(obj):\n",
    "                cars += 1\n",
    "                add_new_object(obj, img, cars)  # Label and start tracking\n",
    "\n",
    "    # Draw the running total of cars in the image in the upper-left corner\n",
    "    cv2.putText(img, 'Cars detected: ' + str(cars), (30, 60),\n",
    "                cv2.FONT_HERSHEY_SIMPLEX, 1.5, DARK_BLUE, 4, cv2.LINE_AA)\n",
    "    # Add note with count of trackers lost\n",
    "    cv2.putText(img, 'Cars lost: ' + str(counters['lost_trackers']), (35, 85),\n",
    "                cv2.FONT_HERSHEY_SIMPLEX, 0.5, DARK_BLUE, 1, cv2.LINE_AA)\n",
    "\n",
    "    cv2.imwrite(\"output/output-\" + filename.split('/')[1], img)\n",
    "    print(\"Processed file {num} of {total_frames}\".format(\n",
    "        num=counters['frames'], total_frames=total_frames), end=\"\\r\")\n",
    "\n",
    "print(\"\\nDone\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Play the annotated frames in the notebook\n",
    "\n",
    "This code will play the annotated frames in a loop to demonstrate the new video.\n",
    "Running this in the notebook is usually slow. Shrinking the size helps some.\n",
    "Refer to the following section to build a real, full speed video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for filename in sorted(glob.glob(os.path.join(os.path.abspath(OUTPUT_DIR),\n",
    "                                              'output-frame_*.jpg'))):\n",
    "    frame = cv2.imread(filename)\n",
    "    clear_output(wait=True)\n",
    "    rows, columns, _channels = frame.shape\n",
    "    frame = cv2.resize(frame, (int(columns/2), int(rows/2)))  # shrink it\n",
    "    _ret, jpg = cv2.imencode('.jpg', frame)\n",
    "    display(Image(data=jpg))\n",
    "\n",
    "print(\"\\nDone\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a video from the annotated frames\n",
    "\n",
    "This command requires `ffmpeg`. It will combine the annotated\n",
    "frames to build an MP4 video which you can play at full speed\n",
    "(the notebook playback above was most likely slow).\n",
    "\n",
    "Uncomment the command to try running it from this notebook, or\n",
    "copy the output files to a system with `ffmpeg` and run the\n",
    "command there.\n",
    "\n",
    "> NOTE: The command below requires libx264 for encoding video\n",
    "stream into the H.264/MPEG-4 AVC compression format. Please \n",
    "check that `ffmpeg` was configured and built with `--enable-libx264`\n",
    "(`ffmpeg 2>&1 | grep libx264`).\n",
    "If not, just remove the `-vcodec libx264` option from the\n",
    "following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !ffmpeg -y -r 60 -f image2 -i output/output-frame_%05d.jpg -vcodec libx264 -crf 25  -pix_fmt yuvj420p annotated_video.mp4\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p><font size=-1 color=gray>\n",
    "&copy; Copyright 2019 IBM Corp. All Rights Reserved.\n",
    "<p>\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file\n",
    "except in compliance with the License. You may obtain a copy of the License at\n",
    "https://www.apache.org/licenses/LICENSE-2.0\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the\n",
    "License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n",
    "express or implied. See the License for the specific language governing permissions and\n",
    "limitations under the License.\n",
    "</font></p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
